tomazbracic
Hi,
I would like to ask for your opinion, find some suggestions on what to use (if there is any existing library I don’t know about), and how to approach it.
I am about to build a new product for parsing Comtrade .cfg and .dat files (there are two more files, but are not mandatory). COMTRADE (Common format for Transient Data Exchange for power systems) is a file format for storing oscillography and status data related to transient power system disturbances .
For better understanding, and if interested into the subject, you can take a look into this to get better understanding
http://dfv-technologie.com/doc_technique_dfv/perturbographie_qualimetrie/Tutorial%20COMTRADE.pdf
For others TLDR; you have .CFG files which describes a structure of .dat files. We’re talking about a mixture of ASCII and BINARY data and so on. Once you create a parser to understand the rules .cfg file define, you can go and parse binary data (.DAT) file. Each event has 1 .cfg and 1 .dat file. Later on I will need to do some correlations between 2 events and so on.
really simplified CFG example
EXAMPLE STATION NO. 1 DAU 32 N,32
15,7A,8D
1,161KV 01-G BUS POTENTIAL,kV, 0.479615,0.0,0.0,-511,511
2,161KV 02-G BUS POTENTIAL,kV, 0.479615,0.0,0.0,-511,511
3,161KV 03-G BUS POTENTIAL,kV, 0.479615,0.0,0.0,-511,511
4,161KV 01 CURRENT, A, 43.187479,0.0,0.0,-511,511
5,161KV 02 CURRENT, A, 43.187479,0.0,0.0,-511,511
6,161KV 03 CURRENT, A, 43.187479,0.0,0.0,-511,511
7,161KV RES CUR., A, 43.187479,0.0,0.0,-511,511
1,PCB 1216 EXAMPLE,1
2,PCB 1226 EXAMPLE,1
3,PCB 1236 EXAMPLE,1
4,161KV CAR. REC,1
5,161KV BUS POTENTIAL UV,1
6,161KV BUS POTENTIAL UF,1
7,EXTERNAL START,1
8,DATA CHECKSUM ERROR N/C,0
60
1
5184,47
01/18/1992,10: 2: 5.136
01/18/1992,10: 2: 5.215
ASCII
####################################################
really simplified DAT example
1,192,252,-233,-19,0,0,0,0,1,1,0,1,1,1,1,0
2,384,243,-243,0,0,0,0,0,1,1,0,1,1,1,1,0
3,576,232,-252,20,0,0,0,0,1,1,0,1,1,1,1,0
4,768,220,-260,40,0,0,0,0,1,1,0,1,1,1,1,0
5,960,207,-267,60,0,0,0,0,1,1,0,1,1,1,1,0
6,1152,192,-273,79,0,0,0,0,1,1,0,1,1,1,1,0
7,1344,176,-276,98,0,0,0,0,1,1,0,1,1,1,1,0
8,1536,160,-279,117,0,0,0,0,1,1,0,1,1,1,1,0
9,1728,142,-280,135,0,0,0,0,1,1,0,1,1,1,1,0
10,1920,124,-279,153,0,0,0,0,1,1,0,1,1,1,1,0
11,2112,105,-277,171,0,0,0,0,1,1,0,1,1,1,1,0
12,2304,85,-274,187,0,0,0,0,1,1,0,1,1,1,1,0
13,2496,65,-269,203,0,0,0,0,1,1,0,1,1,1,1,0
14,2688,45,-263,218,0,0,0,0,1,1,0,1,1,1,1,0
15,2880,25,-255,231,0,0,0,0,1,1,0,1,1,1,1,0
16,3072,5,-246,243,0,0,0,0,1,1,0,1,1,1,1,0
17,3264,-16,-235,253,0,0,0,0,1,1,0,1,1,1,1,0
18,3456,-36,-223,262,0,0,0,0,1,1,0,1,1,1,1,0
19,3648,-56,-210,269,0,0,0,0,1,1,0,1,1,1,1,0
20,3840,-76,-196,276,0,0,0,0,1,1,0,1,1,1,1,0
####################################################
This example above is ASCII type, mine are BINARY
Would you go and implement this on your own from the ground up or you would use some specific (existing) parsing library? I am asking if anybody knows of any library that might help me with parsing itself or if you don’t see this as an issue and would just write your own.
The other still “open” issue on this product is its integration into the Microsoft domain. I assume this shouldn’t be an issue anymore. MS LDAP and other keywords are probably well-supported by now. If you have any thoughts to share more than welcome.
Thanks for your opinion in advance.
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
At WyeWorks, we believe great software is built by combining technical excellence with genuine human connection
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #blog-post
- #elixirconf-us
- #elixir-ls
- #ai
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #hex
- #security
- #metaprogramming
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 4- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
D4no0
One question, is this a structured format or you need some kind of look-ahead to parse it?
If you don’t need look-ahead, you can easily use binary pattern matching to make a very simple declarative parser, I could help if the protocol is not huge.
tomazbracic
Not sure that I fully understand what you mean with look-ahead. Basically, when event is triggered, you get 2 files from that relay HW. 1 cfg file and 1 dat file. First you need to read cfg file in order to know what data is where in your second .dat file. Imagine Utility has like N number of relay in stations and substations. each relay has its own location, number of sensors, basically its own unique configuration. Did I explain?
D4no0
OK, looks like I understand.
In simple database terms the first file defines your “columns” and the second has the actual data AKA “rows”?
D4no0
PM me some sample binary files and I will give it a try.