Network data stream simulation with time range LDA pattern mining -- 2

This project involves the simulation of a SIEM system using Latent Dirichlet Allocation for IoT device streams. It can be implemented in R, Python, C++ or any relevant language that achieves the outcome.

Workflow
Input config > random & pattern generated content streams > stream chunks > LDA parser > output pattern frequency & topics per stream

Data Generation
Input config > random & pattern generated content streams

The generator should be configurable and able to create network simulation data streams. Each stream generates random content and includes generated content as provided by the config file:
1. stream information
2. string and regex patterns to include in the stream (generator fills the regex with matching values)
3. occurrence frequency (range 0 to 10) which represents the number of the generated string and regex patterns to include per minute. Does not have to be very sophisticated, just relatively different.

The generator can be started and stopped.

Example inputs configuration for 2 streams in JSON format.
/ input/config.json
{
{
“name”: “endpoint1”,
“ip”: 10.0.0.8,
“port”: 345,
{
“pattern”: “IP_EXT: '(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3}' MSG: ^#[^ !@#$%^&*(),.?":{}|<>]*$ USER: ^[a-z0-9_-]{3,15}$”
“frequency”: 2
},
{
“pattern”: “PAYLOAD: ^ABC_[^ !@#$%^&*(),.?":{}|<>]*$ ID: ^[a-z0-9_-]{30,150}$”
“frequency”: 5
},
},
{ “name”: “syslog1”,
“ip”: 10.0.0.9,
“port”: 534,

{
“pattern”: “IP_EXT: '(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3}' MSG: ^#[^ !@#$%^&*(),.?":{}|<>]*$ USER: ^[a-z0-9_-]{3,15}$”
“frequency”: 2
},
{
“pattern”: “PAYLOAD: ^ABC_[^ !@#$%^&*(),.?":{}|<>]*$ ID: ^[a-z0-9_-]{30,150}$”
“frequency”: 5
},
},
}

Sample stream chunk.
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed euismod eros a lectus porttitor, vitae aliquet magna ullamcorper. Praesent in enim non magna vehicula faucibus. Vestibulum lacinia velit ut dolor aliquet tincidunt. IP_EXT: 250.209.015.255 MSG: #abyx USER: das-dkjh Ut consectetur hendrerit massa vel tempus. Nulla sit amet libero id felis lacinia accumsan. PAYLOAD: ABC_aS57dasd USR: 42d8ffe6-8a65-416c-ac92-d5826315faa6 In dictum porta magna sed lectus venenatis. Aliquam accumsan molestie augue, sit lectus amet vulputate metus tristique et. Ut a lectus erat elit….

Regex specifications from
https://ihateregex.io/expr/ip/
https://ihateregex.io/expr/username/
https://ihateregex.io/expr/hashtag/


Stream Parser
stream chunks > LDA parser > output pattern frequency & topics per stream

The streams are red by a parser application which reads each input stream for a configurable span of time (e.g. 30 seconds) as input chunks. You must use the Latent Dirichlet Allocation package or method to analyze the data and create/append to 3 log files per stream. Each run is in a new output folder with a timestamp from when the run began.
1. the found matching patterns log (use the input file to identify patterns),
2. the count of the patterns in that timespan log, and
3. up to 10 highest frequency single string terms (LDA topics, occurrence > 1 & not in regex patterns?)

Example
/ output
/ config.json
/ run_timestamp1
/ run_timestamp2
/ run_ 1624313100 # start time of log run
/ endpoint1,
/matches.txt
/ frequency.txt
/ tuning.txt
/ syslog1
/matches.txt
/ frequency.txt
/ tuning.txt

{
“chunk_timespan_seconds”: 30

}


endpoint1/matches.txt (any matched pattern with timestamp)
1624313104: IP_EXT: 250.209.015.255 MSG: #abyx USER: das-dkjh
1624313117: PAYLOAD: ABC_aS57dasd ID: 42d8ffe6-8a65-416c-ac92-d5826315faa6
1624313125: IP_EXT: 250.209.015.254 MSG: #dasdadafg USER: afs-adsfsfsdd
1624313150: IP_EXT: 250.209.015.252 MSG: #dfhdfg USER: sdfff-gdfg

endpoint1/frequency.txt (30 second interval finding summary)