kminevskiy

kminevskiy

Deconstructing CTE into an Ecto query

Hello,

I’m trying to convert a SQL query to the Ecto query, but I’m having hard time figuring out how to do that. I know I could simply execute it using SQL adapter, but I thought I might give it a try using Ecto.

The query is supposed to compare dates in historical_reports table to all non-holiday, non-weekend dates (generated by generate_sequence CTE and then filtered against country_holidays table) since the first date found in the historical_reports table. For example, assuming I have records for a whole year in my historical_reports table and missed yesterday (non-holiday and non-weekend day), the result of executing the following query should be a single record of 2020-04-06.

WITH company_report_date AS
(
	select report_date
	from historical_reports
	where company_id = 1
),
year_without_weekends AS
(
	SELECT date(d)
	FROM generate_series((SELECT min(report_date) FROM company_report_date), current_date, interval '1 day') d
	WHERE EXTRACT('ISODOW' FROM d) < 6
)
SELECT date
FROM year_without_weekends yww
WHERE yww.date NOT IN (SELECT report_date FROM company_report_date) AND
yww.date NOT IN (SELECT holiday_date FROM country_holidays) AND
yww.date <> current_date;

I started deconstructing that CTE into smaller Ecto queries, like so:

company_report_date = from hr in HistoricalReport, where: hr.company_id == ^company_id
first_date_record = from q in company_report_date, select: min(q.report_date)

At this point, I need to use a fragment to generate a sequence of dates, but I’m not sure how to proceed:

from d in fragment("select date(d) from generate_series((?), current_date, '1 day'::interval) d where extract('ISODOW' from d) < 6", first_date_record)

I guess my question is how to construct the third intermediary query when I’m not using any defined schemas / tables and then continue filtering / deconstructing it further. Also, does that approach make sense at all or should I simply use the SQL adapter and run my query as is?

Thanks!

Most Liked

massimo

massimo

I had to do it for a few slightly complex CTEs, your best friend is Ecto.Query.subquery that transforms query composition into subquery expressions.

For example this CTE


with fast as (
	SELECT c2.city, st_distance(st_transform(c2."geom", 4326), point.geom) d 
	FROM "shapes"."cities" AS c2, point 
	WHERE st_intersects(st_transform(c2."geom", 4326), point.geom)
),
slow as (
	SELECT c2.citiy, st_transform(c2."geom", 4326) <-> point.geom d 
	FROM "shapes"."cities" AS c2, point
	WHERE st_transform(c2."geom", 4326) <-> point.geom <= 50.0/110574
)
select * from fast
UNION all
select * from slow
WHERE NOT EXISTS (
	select * from fast
)  
order by d
limit 1

Can be converted to

fast_query =
  from(m in module,
    where: st_intersects(st_transform(m.geom, 4326), st_makepoint(^lon, ^lat)),
    select: map(m, ^fields),
    select_merge: %{
      distance: st_distance(st_transform(m.geom, 4326), st_makepoint(^lon, ^lat))
    }
  )

count_query = from(f in subquery(fast_query), select: %{count: count()})

slow_query =
  from(m in module,
    join: c in subquery(count_query),
    where: in_range(st_transform(m.geom, 4326), st_makepoint(^lon, ^lat), 50),
    select: map(m, ^fields),
    select_merge: %{
      distance: st_distance(st_transform(m.geom, 4326), st_makepoint(^lon, ^lat))
    },
    where: c.count == 0
  )

fast = from(fast in subquery(fast_query))
slow = from(slow in subquery(slow_query))

from(
  q in subquery(
    from(fast in fast,
      union: ^slow,
      select: fast
    )
  ),
  order_by: q.distance,
  limit: 1,
  select: q
)

to print the generated SQL you can use

Ecto.Adapters.SQL.to_sql(:all, RepoModule, query)

EDIT: Ecto 3.2 already added support for CTE, ecto/CHANGELOG.md at master · elixir-ecto/ecto · GitHub but I find subqueries more readable in code, while CTE are usually better in SQL

hauleth

hauleth

Heh, TIL that Ecto finally allows subqueries in the source. Great, I was missing that for some time now.

And to update @kminevskiy - the approach by @massimo will be better as in PostgreSQL 9-11 the CTE was optimisation fence. Now it still is, but it can be lifted sometimes. So this approach will be better as it allows to bubble the filters up and down which allows for better optimisations.

LostKobrakai

LostKobrakai

I guess the problem is you’re using a subquery in the FROM part with generate_series, but ecto queries only support subqueries as from and join sources. Trying to compose subqueries and fragments will quickly become unwieldy, so I’d suggest sticking with the raw SQL unless you can refactor the query to have the generate_series be called differently, so ecto’s subquery handling can apply.

Where Next?

Popular in Questions Top

Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
vac
Hi, I’m quite new in Elixir and I’m trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and I...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
Emily
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I forese...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
komlanvi
Hi everyone, I was playing with phoenix liveView but I run into an issue. I have a form and want to validate each input text when the te...
New
shijith.k
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New
svb
Hi! Currently I want to submit a form by pressing the Enter key. However, since my input field is of type “textarea” this is just adds a...
New

Other popular topics Top

senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
274 41539 114
New
Nvim
Anybody knows a comprehensive comparison of Django and Phoenix, thanks for the help. Where are they similar? Where do they differ the m...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
Lily
In templates/appointment/index.html.eex: &lt;%= for appointment &lt;- @appointments do %&gt; &lt;tr&gt; &lt;td&gt;&lt;%= appoi...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New

We're in Beta

About us Mission Statement