Data visualization with 20mb-500mb trace files
Budget: $750 – $1,500 USD
Project Description:
I am looking for a freelancer who can help me with data visualization. The project involves creating charts and graphs to visualize data from trace files ranging from 20mb to 500mb.
Ideal Skills and Experience:
-***PYTHON ONLY**
- Proficiency in data visualization tools and libraries such as Plotly (but must be web-readable)
- Generate html type pages of the analysis/patterns
- Ability to load and parse large files under 30 minutes for a 500mb file
- Second part of this project is to be able to compare two trace files from dev/prod traces and identify high level differences (additional milestone) so please code with that end game mindset to accept two files -
If you have the necessary skills and experience, please provide examples of your previous work in data visualization.
File notes:
Column 1:53 is static and can be parsed as such into its own buckets with space as the separator.
Column 54:57 will either be "Cur" (sql cursor) and you will have to track sqls with that (#1,#3, all different cursors)
Column 54:57 will either be " >" if it's the start of code
within code logic:
>>> start-ext Nest=2271
will have an ending nest id
<<< end-ext Nest=2271
please track the ending details of:
ResetMessage PTS_SEARCH.BaseSearch.OnExecute Dur=0.001362 CPU=0.000000 Cycles=72
into various buckets
**data such as "Dur=0.001362" should be aggregated into each parent call
**data such as "Cycles=712" should be aggregated into each parent call
parent calls are denoted by 1 less ">"
** See Screenshot attachment for examples of what should be tracked and what can be disregarded **
within sql "Cur" logic:
sample line of:
PSAPPSRV.910 (3369) 1-2260861 08.18.06 0.000109 Cur#1.910.SSCDEV RC=0 Dur=0.000060 COM Stmt=SELECT B.SECURE, A.RECNAME, B.SEARCHRECNAME, C.FIELDNAME, C.FIELDNUM FROM PS_APP_DES_OBJECTS A, PS_APP_DES_OBJ_CST B, PSRECFIELDDB C WHERE A.OBJNAME = :1 AND A.RECNAME = C.RECNAME AND A.PSOWNER = B.PSOWNER AND A.APP_DES_CATEGORY = B.APP_DES_CATEGORY AND A.OBJNAME = B.OBJNAME AND ((ROUND(((C.USEEDIT) / (2)),0) ) * ( 2)) <> C.USEEDIT ORDER BY C.FIELDNUM
Parse: "#1" (cursor number), disregard ".910.SSCDEV RC=0 " (so regex . through space), capture "Dur=0.000060" for your charts, and "COM Stmt=SELECT " will be either =SELECT, =INSERT, =UPDATE, =DELETE for your various chart buckets of SQL statements.
Please capture the actual SQL statement (grab anything after "Stmt=" till new line)
*Disregard anything with "Bind-" in the line.
The following type of lines should be aggregated for each Cur# to capture the "Dur=" for either "Open Cursor" or "Commit" or "Disconnect"
PSAPPSRV.910 (3369) 1-2260892 08.18.06 0.000026 Cur#3.910.notSamTran RC=0 Dur=0.000001 Open Cursor Handle=0000000001269D70
PSAPPSRV.910 (3369) 1-2260893 08.18.06 0.000007 Cur#3.910.SSCDEV RC=0 Dur=0.000000 Commit
PSAPPSRV.910 (3369) 1-2260894 08.18.06 0.000012 Cur#3.910.SSCDEV RC=0 Dur=0.000006 Disconnect
**Please also keep track of the simple COUNT of each one of these calls, so you can tell me as a whole, "We committed 93 times for a total Dur of 2.3 seconds" "We Opened Cursor 2 times for a total Dur of 3 seconds" etc.
I am looking for a freelancer who can help me with data visualization. The project involves creating charts and graphs to visualize data from trace files ranging from 20mb to 500mb.
Ideal Skills and Experience:
-***PYTHON ONLY**
- Proficiency in data visualization tools and libraries such as Plotly (but must be web-readable)
- Generate html type pages of the analysis/patterns
- Ability to load and parse large files under 30 minutes for a 500mb file
- Second part of this project is to be able to compare two trace files from dev/prod traces and identify high level differences (additional milestone) so please code with that end game mindset to accept two files -
If you have the necessary skills and experience, please provide examples of your previous work in data visualization.
File notes:
Column 1:53 is static and can be parsed as such into its own buckets with space as the separator.
Column 54:57 will either be "Cur" (sql cursor) and you will have to track sqls with that (#1,#3, all different cursors)
Column 54:57 will either be " >" if it's the start of code
within code logic:
>>> start-ext Nest=2271
will have an ending nest id
<<< end-ext Nest=2271
please track the ending details of:
ResetMessage PTS_SEARCH.BaseSearch.OnExecute Dur=0.001362 CPU=0.000000 Cycles=72
into various buckets
**data such as "Dur=0.001362" should be aggregated into each parent call
**data such as "Cycles=712" should be aggregated into each parent call
parent calls are denoted by 1 less ">"
** See Screenshot attachment for examples of what should be tracked and what can be disregarded **
within sql "Cur" logic:
sample line of:
PSAPPSRV.910 (3369) 1-2260861 08.18.06 0.000109 Cur#1.910.SSCDEV RC=0 Dur=0.000060 COM Stmt=SELECT B.SECURE, A.RECNAME, B.SEARCHRECNAME, C.FIELDNAME, C.FIELDNUM FROM PS_APP_DES_OBJECTS A, PS_APP_DES_OBJ_CST B, PSRECFIELDDB C WHERE A.OBJNAME = :1 AND A.RECNAME = C.RECNAME AND A.PSOWNER = B.PSOWNER AND A.APP_DES_CATEGORY = B.APP_DES_CATEGORY AND A.OBJNAME = B.OBJNAME AND ((ROUND(((C.USEEDIT) / (2)),0) ) * ( 2)) <> C.USEEDIT ORDER BY C.FIELDNUM
Parse: "#1" (cursor number), disregard ".910.SSCDEV RC=0 " (so regex . through space), capture "Dur=0.000060" for your charts, and "COM Stmt=SELECT " will be either =SELECT, =INSERT, =UPDATE, =DELETE for your various chart buckets of SQL statements.
Please capture the actual SQL statement (grab anything after "Stmt=" till new line)
*Disregard anything with "Bind-" in the line.
The following type of lines should be aggregated for each Cur# to capture the "Dur=" for either "Open Cursor" or "Commit" or "Disconnect"
PSAPPSRV.910 (3369) 1-2260892 08.18.06 0.000026 Cur#3.910.notSamTran RC=0 Dur=0.000001 Open Cursor Handle=0000000001269D70
PSAPPSRV.910 (3369) 1-2260893 08.18.06 0.000007 Cur#3.910.SSCDEV RC=0 Dur=0.000000 Commit
PSAPPSRV.910 (3369) 1-2260894 08.18.06 0.000012 Cur#3.910.SSCDEV RC=0 Dur=0.000006 Disconnect
**Please also keep track of the simple COUNT of each one of these calls, so you can tell me as a whole, "We committed 93 times for a total Dur of 2.3 seconds" "We Opened Cursor 2 times for a total Dur of 3 seconds" etc.