Let\u2019s start by taking the following log-entry from my logentries\u2019 account. I have changed the URL randomly. All nginx access logs would be coming in this format only as this format will not change until I change it. We would be creating our KVP on basis on this format.<\/p>\n
[js]2015-05-27T08:34:01.557805Z web-server1 nginx-access-log – – – hostname=web-server1 appname=nginx-access-log 0.350 117.247.13.187 – – [27\/May\/2015:14:04:01 +0530] "GET \/undefined\/uekf=12ed1 HTTP\/1.1" 404 14657 "http:\/\/www.mydomain.com\/?abc=xyz"[\/js]<\/p>\n
Now we are going to create RegEx named capture group which is nothing but creating KVP dynamically.
\nSyntax of RegEx named capture group is as below.<\/p>\n
[js]\/some anchor text to finds key location in log (?P<KEY>regEx_to_find_the_Value)\/[\/js]<\/p>\n
Here whatever the \u201cregEx_to_find_the_Value\u201d would return is assigned to \u201cKEY\u201d.
\n\u201cregEx_to_find_the_Value\u201d should evaluate to url which is in this particular case is \u201c\/undefined\/uekf=12ed1\u201d and this could be any url.<\/p>\n
In the log, we have url as \u201c\/undefined\/uekf=12ed1\u201d, \u2018”GET \u2018 which is present just before url can be taken as anchor text. We can use any string as as key, let it be “URL”. Now we are required to define RegEx for the urls. Now urls can contains words, numbers, spaces so we have to define regex accordingly. [js]\/"GET (?P<URL>[\\\/\\w\\d\\s.]+)\/[\/js]<\/p>\n Now we are ready with dynamic KVP. [js]404 AND \/"GET (?P<URL>[\\\/\\w\\d\\s.]+)\/ groupby(URL) calculate(count)[\/js]<\/p>\n Here we are using \u201cAND\u201d, groupby and calculate functions provided by logentries.<\/p>\n Below is the graphical view which I get on searching with above expression in logentries. Reference: Thanks, Most of us would be using some tools for centralizing logs, their analysis and storage. Logentries is also the one falling into this category. Logentries provides a straightforward way for analysis of logs containing KVPs i.e. Key-Value pairs. But for the cases where KVPs are not present it becomes quite hectic to analyse the logs. […]<\/p>\n","protected":false},"author":154,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":6},"categories":[1174],"tags":[1783,431],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/20855"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/154"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=20855"}],"version-history":[{"count":0,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/20855\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=20855"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=20855"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=20855"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\nSo, the regex would be \u201c[\\\/\\w\\d\\s.]+<\/strong>\u201d.
\nwhere \u201c\\\u201d is escape character for \u201c\/\u201d
\n\\w denotes word,
\n\\s denotes space,
\n\\d denotes digits,
\n. denotes any character other than newline.
\nThe [ ] brackets mean anything inside the square brackets will be checked in the regEx.
\n\u201c+\u201d denotes the regEx can occur one or more times.
\nWe have created the following regEx till now.<\/p>\n
\nAs per our scenario, we want all urls throwing 404 error so we first need to search 404. Then we will apply the RegEx named capture group on this result. We need to group the same urls and calculate the occurrence of same urls.SO the final expression will come out to be as follows.<\/p>\n
\n![\"log-blog\"](\"\/blog\/wp-ttn-blog\/uploads\/2015\/06\/log-blog.png\")
<\/p>\n
\nhttps:\/\/logentries.com\/doc\/regex\/#regex-field-extraction
\nhttps:\/\/github.com\/logentries\/le\/blob\/master\/README.md#follow-log-files-through-your-configuration-file
\nhttp:\/\/www.regexr.com\/<\/p>\n
\nNavjot Singh
\nTeam AWS, TO THE NEW Digital.
\nnavjot[dot]singh[at]tothenew[dot]com<\/p>\n","protected":false},"excerpt":{"rendered":"