I had trouble with multiple time frames on tradingview’s pine script, and data munging was a pain (for python libraries) so I built myself a simple backtest engine too. It just reads ohlc from postgres and crunches the numbers, I ended up with a module with 1k+ lines of code… The whole thing is slightly better than excel because other parts of my engine has code to pull data from oanda and push alerts to slack. Personally I think it’s unrealistic to rely on a trading bot, the best a bot can do is scan things for you based on some decision tree but that’s out of the scope of this discussion.
xlphs
CinderellaMan
Hello everybody,
@rm-rf-etc sorry for a late reply - I was away from a computer for a couple of days, but I’ve seen your message and have been thinking about it.
Regards abstractions and trying to squeeze in everything into a single system - partly I would agree that some things could be generalized for example “tick” prices/candles, etc for the display purposes - crypto, forex, stocks could just have price and time fields and that’s about it - so really simplified no generalized.
From everything that I’ve done over the years, I think the system needs to be really simple to succeed. People will build on top of it so it needs to be as simple as this is the data and you do whatever you want with it (write your strategy). In my course (which looks already much different than Igthorn: GitHub - Cinderella-Man/hedgehog at video_07 · GitHub [at this moment at the 7th episode]) I took a PubSub approach. If you think about it - the only things that consumer needs to know is the data struct that will be broadcasted and the topic name. It works for anything that you want to stream - just prefix your stuff with type of what you stream. You would have topics like “stocks:AAPL”, “forex:GBPUSD”, “crypto:ADAUSDT” - nothing would interfere with each other. Then you write your specific consumers for crypto data, forex data, or stock data. There will be multiple consumers for single topic - one storing data to Postgres (or timescaleDB - or both[would guess that would be 2 consumers though!] as this could be beneficial as well!), one making some intermediate indicators and publishing them forward to topics like “forex:GBPUSD:ATR” or something like that. Obviously one of the consumers will be your strategy.
The above idea works perfectly with backtesting as you can just open the database (or a file) and publish everything to the topic and in the end, you will have a database with all orders, transactions and anything else that would happen.
I used Postgres as I don’t have any experience with timescaleDB. I would shy away from dropping too
much tech if there’s no extremely good reason for it.
I know of one more trading project from @rupurt - GitHub - fremantle-industries/workbench: From Idea to Execution - Manage your trading operation across a distributed cluster · GitHub
I didn’t look into it but it’s also crypto-related and it looks like it’s more ‘abstracted’ away as it marketed as a workbench to create strategies on top of.
So in my opinion, it’s more the idea to embrace the common core that will just contain the project structure, possibly a frontend (so also it will need to contain common simplified structures), probably a bunch of behaviors that will dictate what is expected from each element of the system like “DataStreamer” - it needs to have “switchOn”, “switchOff” methods (that is just an example).
I would be interested in planning this out but I need to admit that I’m starting new work on Monday and also record my YouTube videos so I’m a little bit low on time. I need to give a good look at all of your projects (the ones that are shared on GitHub) and compare - I would love to brainstorm and come up with something as I can see that we are all doing similar things and wasting time by reinventing the wheel.
I wanted to keep this short… failed again - sorry
0
rm-rf-etc
Welcome back 
. I can give a longer response later, gotta make this one brief.
TimescaleDB is a postgres extension / plugin / add-on, whatever it’s called, so you get all the goodness of ecto, but optimized for time series data.
Glad we’re all thinking generally the same way! This is exciting! And yes, I think we should get some documentation together to organize planning.
@GunnarPDX from our DM chat, here’s my first pass on pattern detection:
https://github.com/rm-rf-etc/ohlc-pattern-detection
The approach is to (1) extract the reversals from the chart (the maximum highs and lows forming a zig-zag through the chart), then (2) normalize the set, and finally (3) test the set with some rules. I’ve only gotten as far as steps 1 and 2. My thinking is based on the books by Thomas Bulkowsky and what I’ve read of his method on his website, as well as a Medium post which described something very similar.
Also, this approach assumes you will use E.take/2 to grab your maximum size sample from your chart, and then you iterate over the reversals, continually dropping 1 off the beginning in each iteration, then normalize, and then test again. This way you will be scanning for a pattern of any length between your minimum and your maximum length.
0
rm-rf-etc
Anybody want to write an implementation of this one?
https://towardsdatascience.com/detection-of-price-support-and-resistance-levels-in-python-baedc44c34c9?gi=530b2ea158f9
I would change one thing about it though. I would calculate the support and resistance as areas.
0
GunnarPDX
I could work on that one. What data source are you using? Is the data just formatted as a list with {high, low, open, close} for time intervals?
0
rm-rf-etc
Sometimes I just read them from a chart in TradingView and type it out manually, which is slow. Other times, I’ll use Webull and some code I’ve written to pull from their API, which is entirely open, just not documented. Here’s an endpoint you can use:
You’ll need to figure out the timestamp. I have some code to help with that but it’s not yet public. I will publish in when I get a chance. You’ll need to get the stock ID. Here are a few you could use:
MSFT: 913323997
AAPL: 913256135
TSLA: 913255598
AMZN: 913256180
NFLX: 913257027
FB: 913303928
Each bar is a string, which you can parse with this function:
def parse_bar(str) when is_binary(str) do
[ts, o, c, h, l, _ma, v, vwap] = String.split(str, ",") |> Enum.take(8)
%{
t: if(not_null(ts), do: String.to_integer(ts)),
o: if(not_null(o), do: String.to_float(o)),
c: if(not_null(c), do: String.to_float(c)),
h: if(not_null(h), do: String.to_float(h)),
l: if(not_null(l), do: String.to_float(l)),
v: if(not_null(v), do: String.to_integer(v)),
vw: if(not_null(vwap), do: String.to_float(vwap))
}
end
0
rm-rf-etc
Thanks @GunnarPDX, that was quick. Looking forward to testing it when I have some time (probably this weekend).
I’ve been thinking about this quite a bit. I think this is the right idea. Although I’m not sure how I feel about the indicators going to topics. The way I’ve approached it is that I have a state struct that contains the OHLC price chart and the output from whatever indicators we want to use:
%LiveTrade{
charts: %{
m1: %{
max_list_length: 200,
bars: [],
macd: %Indicators.MACD{}
rsi: %Indicators.RSI{}
... any others you want here ...
},
m5: %{
max_list_length: 40,
bars: [],
macd: %Indicators.MACD{}
rsi: %Indicators.RSI{}
... any others you want here ...
}
}
}
So with this, the timeframe of the price chart is inherently the same for any of the indicators attached. :charts can contain any number of timeframes we want, as the module just loops over all the keys under :charts. So the whole system works as a state machine, the previous state is passed into the first function along with the new candlestick object, the function merges it into the chart, and then this state object is piped into each indicator function which then generates the updated indicator list and writes it to the state struct.
So I’m thinking about a few things from here.
- The chart
max_list_lengthis used to keep the list from getting too large for memory, so the data then needs to be written to the DB, and I haven’t worked on that piece yet. - Sometimes some of the real-time data shows up after a delay (https://polygon.io/blog/aggregate-bar-delays/) and for some use cases, we want to update bars that have already been on the chart for a while. In this case, we would have to basically rollback to the part of the chart which is going to be modified, merge the delayed data in, and then iterate forward until we’re back to the latest OHLC packet. I haven’t worked on this problem yet either.
- In the case of backtesting, we have to make sure we have all the necessary data in the DB before we start. In some cases, we may have parts of the data and gaps in between, so we would want to only download the data needed for filling the gaps, and then allow the backtest to run. So I’m thinking about how we might solve this with a PubSub approach. And I’m also thinking about how it would work if built on just
StreamandTask.async. Perhaps the primitive would beStreamobjects, and then the layer above could implement the PubSub parts.
0
rm-rf-etc
I think I want to drop use of timescaleDB. The following article advocates using SQLite:
I use SQLite every time it is possible because it is extremely simple and easy to use and backup. Despite many people state that it is a too basic database it can handle huge large sets. Its major drawback is its lack of page or area locking — which leads to performance issues in applications that write a lot — . Other than that, it usually performs much better than expected, it is really lightweight and it has zero maintenance.
https://towardsdatascience.com/how-to-store-financial-market-data-for-backtesting-84b95fc016fc
Since handling market data will be light on updates and heavy on reads, this idea seems reasonable to me. But, I’m not a data engineer, so if anybody here is able to lend some wisdom, I would greatly appreciate it.
0
