Hi friends, I’m totally newbie, but since I see this error I can’t figure out what if causing this error.
This error triggered on a recurrent cron task which checks db for flag then script gets token to fetch data from google maps API.
Some tokens are going well, and script successfully fetched and stored data in DB, some with such error fails and stops script. Error stays the same until i delete unprocessible data.
The only tokens are changing and used to fetch data from google maps api.
To stop this error I have to go do db and delete unwanted data which can’t be processed due unknown (for me) issue.
Where to go to check the initial reason why this failures appear?
2022-10-20 23:30:01.455 [error] Handle geo point error - %BadMapError{term: nil}, geo - %Schema.Geo{__meta__: #Ecto.Schema.Metadata<:loaded, "geos">, ancestry: nil, ancestry_depth: 0, awards: #Ecto.Association.NotLoaded<association :awards is not loaded>, companies: #Ecto.Association.NotLoaded<association :companies is not loaded>, country_info: #Ecto.Association.NotLoaded<association :country_info is not loaded>, country_info_id: nil, created_at: ~N[2022-10-20 21:15:44.996481], full_name: nil, groups: #Ecto.Association.NotLoaded<association :groups is not loaded>, id: "18739c62-a8f5-4528-ad27-5610d30be8e4", kind: "latlng", latitude: 36.627206160762825, legacy_id: 3058, longitude: 31.76591573513031, name: nil, state: "draft", translations: nil, units: #Ecto.Association.NotLoaded<association :units is not loaded>, updated_at: ~N[2022-10-20 21:15:44.996481]}
the db record of an affecting entry is this
and the kind is a flag and latitude+longitude (geopoint) is the token to fetch data
After the script is rightly finished, it fills changes the flag to type localities fetched from g.api data, and is translations and puts in a translations jsonb with fetched translations for different langs.
Firstly script finds the country associated with geopoint, then checks if same data collected earlier and deletes this entry if duplicate found, then updates the id in related tables to that one which is already in db.
If no dups found it fetches data from google map API and builds multiple records, yet also checks for dups before creation, then fills the ancestry so keep the logic of google API in our proxy geo db where we store different languages needed to show the data w/o need to fetch every time google api.
part of schema in fetching script
1 defmodule Mix.Tasks.Geos.CompleteAncestry do
2 use Mix.Task
3 import Ecto.Query, only: [from: 2]
4 require Logger
5 require IEx
6
7 @api_key "APITOKEN"
8 @domain "https://maps.googleapis.com/maps/api/geocode/json"
9 @localities [
10 %{external_name: "country", name: "country"},
11 %{external_name: "administrative_area_level_1", name: "division"},
12 %{external_name: "administrative_area_level_2", name: "region"},
13 %{external_name: "administrative_area_level_3", name: "subregion"},
14 %{external_name: "administrative_area_level_4", name: "settlement"},
15 %{external_name: "administrative_area_level_5", name: "village"},
16 %{external_name: "locality", name: "city"},
17 %{external_name: "sublocality_level_1", name: "district"},
18 %{external_name: "sublocality_level_2", name: "locality"},
19 %{external_name: "sublocality_level_3", name: "commune"},
20 %{external_name: "sublocality_level_4", name: "commune1"},
21 %{external_name: "sublocality_level_5", name: "commune2"},
22 %{external_name: "establishment", name: "establishment"},
23 %{external_name: "street_address", name: "street_address"},
24 %{external_name: "neighborhood", name: "neighborhood"}
25 ]
26
27 def run(args) do
28 Application.ensure_all_started(:platform)
29 time = DateTime.from_unix!(DateTime.to_unix(Timex.now()) - 5 * 60)
30 from_time = DateTime.from_unix!(DateTime.to_unix(Timex.now()) - 15 * 60)
31 Logger.info("Start complete ancestry time last - #{time}")
32 # https://developers.google.com/maps/faq#languagesupport
33 # https://developers.google.com/maps/documentation/geocoding/requests-geocoding#language-param
34 # we may hit usage limits on number multiplied by locales > 5 https://developers.google.com/maps/faq?hl=en#usage-limits
35 # 50 QPS is a limit, at max we fetch for 14th languages what seems to be 14 QPS
36 Enum.each(get_points(args, time, from_time), fn record ->
37 try do
38 en = request_by_point(record, "en")
39 de = request_by_point(record, "de")
[cutted]
52 handle_localities(%{en: en, de: de, sv: sv, fi: fi, da: da, nl: nl, fr: fr, it: it, es: es, pt: pt, ru: ru, bg: bg, el: el, tr: tr}, record)
53 rescue
54 e ->
55 Logger.error("Handle geo point error - #{inspect(e)}, geo - #{inspect(record)}")
56 end
57 end)
58 end
59
60 def get_points(args, time, from_time) do
61 cond do
62 is_binary(args) && args == "old" -> Schema.Geo.get_old_by_latlng(time, from_time)
63 true -> Schema.Geo.get_last_by_latlng(time)
64 end
65 end
[cutted]
123 def update_last_geo(record, point, country_info_id) do
124 cond do
125 point && point.geo && !point.exist ->
126 # if point geo not exist in database, update geo and delete last point
127 record
128 |> Schema.Geo.changeset(%{
129 kind: point.geo.kind,
130 state: point.geo.state,
131 ancestry_depth: point.geo.ancestry_depth,
132 ancestry: point.geo.ancestry,
133 name: point.geo.name,
134 full_name: point.geo.full_name,
135 country_info_id: country_info_id,
136 translations: point.geo.translations
137 })
138 |> Platform.Repo.update!()
139
140 update_geom(record)
141
142 Logger.info("Deleted temporary geo - #{inspect(point.geo)}")
143 point.geo |> Platform.Repo.delete()
144
145 point && point.geo && point.exist ->
146 # if point geo exist in database, update all points in Units, Groups, CompanyOffices
147 update_relations(record.id, point.geo)
148
149 Logger.info("Deleted exist geo - #{inspect(record)}")
150 record |> Platform.Repo.delete()
151
152 true ->
153 Logger.info("Bad data for geo #{inspect(record)}")
154 end
155 end
[cutted]
212 cond do
213 geo ->
214 {geo, true}
215
216 true ->
217 country_info = List.first(ancestries).country_info
218 geo =
219 %Schema.Geo{}
220 |> Schema.Geo.changeset(%{
221 kind: kind,
222 state: state,
223 latitude: location["lat"],
224 longitude: location["lng"],
225 ancestry_depth: length(ancestries),
226 ancestry: Enum.map_join(ancestries, "/", & &1.id),
227 name: name,
228 full_name: full_name,
229 country_info_id: (if country_info, do: country_info.id, else: nil),
230 translations: %{
231 en: %{name: name, fullname: full_name},
232 de: %{
233 name: build_name(result.de),
234 fullname: build_full_name(ancestries, build_name(result.de), :de)
283 },
[cutted]
284 }
285 })
286 |> Platform.Repo.insert!()
287
288 update_geom(geo)
289 {geo, false}
290 end
291 end
[cutted]
