Added additional env vars to control the frequency of detector activating - #33
Added additional env vars to control the frequency of detector activating#33symstu-tempesta wants to merge 2 commits into
Conversation
aca57a6 to
f73cced
Compare
krizhanovsky
left a comment
krizhanovsky
left a comment
There was a problem hiding this comment.
I have a couple of comments, where I think we should do different things
00cd8b4 to
2e17cd3
Compare
krizhanovsky
left a comment
krizhanovsky
left a comment
There was a problem hiding this comment.
There are still questions on the implementation
| BLOCKING_NEW_WINDOW_OFFSET_SEC=10 | ||
|
|
||
| # Duration (in seconds) of the **new** (current) window. | ||
| BLOCKING_NEW_WINDOW_DURATION_SEC=10 |
There was a problem hiding this comment.
Why do we need these 4 new parameters? #25 requires only 2 new ones: BLOCKING_CHECK_TIMEOUT_SEC and TRAINING_MODE_HISTORY_OFFSET_MIN.
Also please write more detailed commit messages. E.g. 3890839 provides only a brief "additional variable to control data collecting intervals" and doesn't answer why do we need these 4 new parameters. There are also no comments in the code or config example, which also describes the parameters as "new" and "previous" but doesn't explain when and why the should be configured.
There was a problem hiding this comment.
I was planning to squash all commits later and add a more detailed description there, as this is a continuation of the changes and it's much easier to track them this way. The last changes I've made according to the manual on our web-site.
The example we discussed included gaps between windows — for example, the same day one week ago. In that case, we need to control the gap duration, the window duration, and its start time. While working with these parameters, I realized it would also be useful to give maximum control over the windows and make it more straightforward.
Now it is possible to configure each window independently, including its start time and duration — for example, a window that started 2 hour ago and lasts for 1 hour, and another window that started a week ago and lasts for 1 day.
There was a problem hiding this comment.
OK, now is 2pm and we have 2 windows: [10am-11am] and [12pm-1pm], which means that we analyze data 2-1 hours ago and validate the model against the data for 4-3 hours ago. Now we can say that there was an attack 1 hour ago. What's the purpose of the automated daemon if we can run the queries about the very past events manually? This is simple forensic discovery.
There was a problem hiding this comment.
The question is reasonable. I believe it could be helpful for testing to compare some datasets. For instance, we know that a DDoS attack happened yesterday and want to check whether the app could detect it — in this case, we can start the app on old data to verify the incident. By default, the newest window has a zero value and starts from "now", so you only need to set 3 variables
8a8fe59 to
9f85301
Compare
It is now possible to set the frequency of fetching data from ClickHouse and configure the fetching queries more accurately. For instance, it's good to fetch data for 1 day from a day before and one week before. #25
Google has migrated the list of crawlers to the new url addresses
9f85301 to
1c3efe1
Compare
krizhanovsky
left a comment
krizhanovsky
left a comment
There was a problem hiding this comment.
There are no much changes since the last review
3 participants
No description provided.