text file to 'other' sources

[audi321]

Hi all, I'm looking for someone to assist with a mini project and normally I ask for 'free' advice, but this time I've some funds available to pay someone.

Basically, I have a text file (.log) which contains a load of data (gobblediguke to me) and is updated in real time, which I need to have some kind of dashboard created from it (not bothered how pretty it is) but I need to report from this text file in real time.

Not a huge task for the right person I know, but if you've the skills to assist, and want some weekend tokens, please pm me.

[jjones]

have a look at nagios

[fat80b]

doesn't sound like Nagios is quite right from the description to me.

To me I would have thought a bit of python, a simple database and a grafana front end as a dashboard would be pretty straightforward to implement

[Lynx516]

audi321 said:

Hi all, I'm looking for someone to assist with a mini project and normally I ask for 'free' advice, but this time I've some funds available to pay someone.

Basically, I have a text file (.log) which contains a load of data (gobblediguke to me) and is updated in real time, which I need to have some kind of dashboard created from it (not bothered how pretty it is) but I need to report from this text file in real time.

Not a huge task for the right person I know, but if you've the skills to assist, and want some weekend tokens, please pm me.

Pretty sure this is exactly what Elastic’s logstash and Kibana were made for.

[nyt]

Also check out Splunk - free for smaller installs

[camel_landy]

nyt said:

Also check out Splunk - free for smaller installs

^^^ This...

There's a rather handy free introductory course they offer too:

https://www.splunk.com/en_us/training/free-courses...

Logstash, Elastics, etc... are another alternatives but can be a bit fiddly to get going. Nagios is more about alerting on events & failures.

HTH

M

[Stupeo]

You need the ELK stack: https://www.elastic.co/what-is/elk-stack

[judas]

logz.io is a SaaS ELK stack. We're using at the moment to analyse log files. The complication is more likely to be getting the log files into it if you're not tech-savvy.

What kind of log files are they and what sort of analysis do you need to run?

[audi321]

Thanks for the replies guys, but I'm probably not able to do any of these things myself due to lack of skills, hence asking for someone to do it and I'll pay them.

The log file contains data from a system we use and it's nothing complicated (DTS, url, transaction type, amount paid, etc etc), but I need to convert it to some kind of pretty output to show management how many of each particular line has been transacted during that period (if easier I can take copy snapshots from the log file manually each period as it's updated in real time so the file cannot be opened).

The only real complexity in the log file is that there's some base64 decoding to do to read some of the fields.

[paulrockliffe]

If you can take copies then you can might be able to convince Excel to dashboard it. (2016 onwards)

Click on the Data tab on the Ribbon, then Get Data -> From Text File and see how it gets on. It'll load the file into Power Query and from there you can quickly build a process to convert it into a data table. It sounds like your data would be in a format that could work, but of course it's impossible to say without looking at it.

When you're done, Close and Load the Query into the Data Model and it'll be available to you when you add Pivot Tables and Charts.

Not sure how Excel would work with a 'live' file, I've not tried that, so I don't know that it won't work.

[audi321]

Ah yes, I am able to get it nicely into excel using that method.

The issue I have is that a lot of the data I need to report on is within this Base64 encoded cell (and it's massive, much longer than the max 255 width of an excel cell).

[Lynx516]

audi321 said:

Thanks for the replies guys, but I'm probably not able to do any of these things myself due to lack of skills, hence asking for someone to do it and I'll pay them.

The log file contains data from a system we use and it's nothing complicated (DTS, url, transaction type, amount paid, etc etc), but I need to convert it to some kind of pretty output to show management how many of each particular line has been transacted during that period (if easier I can take copy snapshots from the log file manually each period as it's updated in real time so the file cannot be opened).

The only real complexity in the log file is that there's some base64 decoding to do to read some of the fields.

I suppose the question is how much data are we talking about? My suggestion and a few others of the ELK stack is based of a decent amount of data being updated every second or more frequently. If its only say 100mb and updated once a day then something like PowerBI or Tableau should work, they should be able to refresh off the file.

Getting someone to implement Elastic is going to be expensive, but PowerBI/Tableau is pretty user friendly and yo may be able to to yourself, if not it would be a lot cheaper!

[audi321]

There's not much data at all! Like I say it's a mini project and the log text file is under 1mb

[Lynx516]

PowerBI or similar should work then, as it should do the Base64 decode without a length limit. If you have someone who knows python then something like a Jupiter notebook might also suit.

[audi321]

Lynx516 said:

PowerBI or similar should work then, as it should do the Base64 decode without a length limit. If you have someone who knows python then something like a Jupiter notebook might also suit.

That's my point.....I don't have anyone, hence asking on here. I've a bit of budget from work and I thought if someone on here wanted some weekend tokens then feel free to pm me, but no one has lol. You're either all too busy or have too much ££££ lol.

[paulrockliffe]

audi321 said:

Ah yes, I am able to get it nicely into excel using that method.

The issue I have is that a lot of the data I need to report on is within this Base64 encoded cell (and it's massive, much longer than the max 255 width of an excel cell).

I don't know if you edited and added that requirement in or I just missed it, but that's the only bit I'm not sure about with Excel. I'm not saying Excel is the best tool for this though, just offering it as you'll obviously have it and chances are if you know enough to ask the question, you'll know your way around it.

Anyway, you can sidestep that cell limit by not putting the result into a cell. Load to the Data Model and there's no cell limit. Put your output into a Pivot Table based on the Data Model and you'll be able to display the text.

The tricky bit is that you'll need to unencode it in Power Query. I don't know anything about unencoding Base64, but I'm going to guess it's possible. If you can apply the unencode as a rule to the cell content then you'll be able to do it in Power Query, either by defining the rule or calling a regular expression or something along those lines. Maybe.

[geeks]

audi321 said:

Lynx516 said:

PowerBI or similar should work then, as it should do the Base64 decode without a length limit. If you have someone who knows python then something like a Jupiter notebook might also suit.

That's my point.....I don't have anyone, hence asking on here. I've a bit of budget from work and I thought if someone on here wanted some weekend tokens then feel free to pm me, but no one has lol. You're either all too busy or have too much ££££ lol.

I can recommend a guy if that helps?

https://www.weavingwebs.co.uk/ ask for Simon, tell him Dan sent you.

[paulrockliffe]

I Googled the Base64 thing, this is for PowerBI, but it's functionally the same in Excel PowerQuery where you can use M and DAX in the same way that you can in PowerBI.

https://community.powerbi.com/t5/Desktop/Is-there-...

Looks pretty straightforward.

[maffski]

If it's for an occasional snapshot can you throw the base64 column(s) through an online tool? something like https://onlinecsvtools.com/convert-base64-to-csv

[audi321]

I’ve found a VBA for decoding the base64 in Excel so I’m actually getting there myself in Excel thanks to your advice to date Paul with the PowerQuery. I’ve not used this before so I’m just doing some YouTube training to bring myself up to date before going further.

Would prefer to stay in something familiar to me like Excel.