Size: a a a

ClickHouse не тормозит

2020 June 08

DC

Denny Crane (I don't... in ClickHouse не тормозит
DimON
А там можно вставить что-то вида:
JSONExtractBool(JSONExtractString(DATA,'F1'),'F2',F3'))
ну да, я это и предложил сделать
источник

DC

Denny Crane (I don't... in ClickHouse не тормозит
Piotr Liakhavets
на входе unixtime, прилетело некорректное ~64000
через toDate(64000) →
SELECT toDate(64000)

┌─toDate(64000)─┐
│    2106-02-07 │
└───────────────┘
show create table можете показать?
источник

D

DimON in ClickHouse не тормозит
Denny Crane (I don't work at Yandex (never did))
ну да, я это и предложил сделать
Спасибо. Из документации не понял глубину кроличьей норы. Пошел пробовать.
источник

PL

Piotr Liakhavets in ClickHouse не тормозит
Denny Crane (I don't work at Yandex (never did))
.bin файлы в папке distributed таблицы
заглянул - орда (13к) файлов .bin
это залипшие последующие инсерты?
или только с эксепшеном что свалилось?
источник

DC

Denny Crane (I don't... in ClickHouse не тормозит
Piotr Liakhavets
заглянул - орда (13к) файлов .bin
это залипшие последующие инсерты?
или только с эксепшеном что свалилось?
ну в логе КХ все ответы, мне его не видно
источник

PL

Piotr Liakhavets in ClickHouse не тормозит
Denny Crane (I don't work at Yandex (never did))
show create table можете показать?
CREATE TABLE lz.braingame_android_distributed_bigbase12_ets_events (`created_at_dt` Date, `file_dt` Date, `file_hr` UInt8, `created_at` UInt32, `installed_at` UInt32, `adid` FixedString(16), `idfa` FixedString(16), `idfv` FixedString(16), `gps_adid` FixedString(16), `city` String, `country_subdivision` String, `postal_code` String, `session_count` String, `install_finish_time` String, `install_begin_time` String, `app_id` String, `app_name` String, `app_version` String, `app_version_raw` String, `network_name` String, `campaign_name` String, `adgroup_name` String, `creative_name` String, `label` String, `is_organic` String, `gclid` String, `click_referer` String, `android_id` String, `referrer` String, `search_term` String, `country` String, `device_name` String, `device_type` String, `os_name` String, `api_level` String, `sdk_version` String, `os_version` String, `revenue_usd` String, `match_type` String, `activity_kind` String, `event_name` String, `tracker` String, `fb_campaign_group_name` String, `fb_campaign_name` String, `fb_adgroup_name` String, `tweet_id` String, `twitter_line_item_id` String, `connection_type` String, `isp` String, `conversion_duration` String, `last_time_spent` String, `time_spent` String, `lifetime_session_count` String, `source` String, `balance_ver` String, `game_count` String, `time_s` String, `score` String, `block_count` String, `lines` String, `squares` String, `best_score` String, `miss_block0` String, `miss_block1` String, `miss_block2` String, `stage` String, `hints` String, `block_value` String, `remove_ad` String, `count` String, `action` String, `skin_id` String, `prv_skin_id` String, `date` String, `time_05s` String, `day` String, `text` String, `placement` String, `place` String, `networkName` String, `creativeId` String, `orientation` String, `mode` String, `time_1s` String, `line_item_1` String, `line_item_2` String, `type` String, `viewCount` String, `id` String, `reason` String, `connection` String, `screen` String, `app` String, `time_01s` String, `statusCode` String, `errorCode` String, `button` String, `eu` String, `issue` String, `per_user` String, `per_session` String, `interval_renamed` String, `ignore_in_a_row` String, `seq_num` String, `mopub_ilrd_adgroup_id` String, `mopub_ilrd_adgroup_name` String, `mopub_ilrd_adgroup_priority` String, `mopub_ilrd_adgroup_type` String, `mopub_ilrd_adunit_format` String, `mopub_ilrd_adunit_id` String, `mopub_ilrd_adunit_name` String, `mopub_ilrd_country` String, `mopub_ilrd_currency` String, `mopub_ilrd_id` String, `mopub_ilrd_network_name` String, `mopub_ilrd_network_placement_id` String, `mopub_ilrd_precision` String, `mopub_ilrd_publisher_revenue` String, `rewarded` String, `session` String, `applies` String, `lat` String, `personalized_ads` String, `consent_ads_state` String, `consent_easy_state` String, `option_state` String, `pages` String, `region_detection` String, `link` String, `banner_request` String, `banner_loaded` String, `banner_failed` String, `inter_request` String, `inter_loaded` String, `inter_failed` String, `rewarded_request` String, `rewarded_loaded` String, `rewarded_failed` String, `name` String, `level_progress` String, `state` String, `level_id` String, `from` String, `hints_unlock` String, `setting` String, `change_count` String, `tasks` String, `level_number` String, `content_version` String, `download_error` String, `middle` String, `value` String, `balance_version` String, `skip` String, `clickTrackingUrl` String, `test` String, `group` String, `fromScrn` String, `fromCtrl` String, `slideOn` String, `slideTo` String, `monthDif` String, `by` String, `dayDif` String, `month` String, `progress` String, `available` String, `userLvl` String, `cnt` String, `year` String, `game_mode` String, `main` String, `daily` String, `consent_ccpa_state` String, `created_at_micro` UInt64, `source_file_name_for_debug` String, `timezone` String, `impression_id` String, `waterfall` String) ENGINE = Distributed('bigbase12', 'lz', 'braingame_android_shard_bigbase12_ets_events', sipHash64(adid))
источник

DC

Denny Crane (I don't... in ClickHouse не тормозит
Piotr Liakhavets
CREATE TABLE lz.braingame_android_distributed_bigbase12_ets_events (`created_at_dt` Date, `file_dt` Date, `file_hr` UInt8, `created_at` UInt32, `installed_at` UInt32, `adid` FixedString(16), `idfa` FixedString(16), `idfv` FixedString(16), `gps_adid` FixedString(16), `city` String, `country_subdivision` String, `postal_code` String, `session_count` String, `install_finish_time` String, `install_begin_time` String, `app_id` String, `app_name` String, `app_version` String, `app_version_raw` String, `network_name` String, `campaign_name` String, `adgroup_name` String, `creative_name` String, `label` String, `is_organic` String, `gclid` String, `click_referer` String, `android_id` String, `referrer` String, `search_term` String, `country` String, `device_name` String, `device_type` String, `os_name` String, `api_level` String, `sdk_version` String, `os_version` String, `revenue_usd` String, `match_type` String, `activity_kind` String, `event_name` String, `tracker` String, `fb_campaign_group_name` String, `fb_campaign_name` String, `fb_adgroup_name` String, `tweet_id` String, `twitter_line_item_id` String, `connection_type` String, `isp` String, `conversion_duration` String, `last_time_spent` String, `time_spent` String, `lifetime_session_count` String, `source` String, `balance_ver` String, `game_count` String, `time_s` String, `score` String, `block_count` String, `lines` String, `squares` String, `best_score` String, `miss_block0` String, `miss_block1` String, `miss_block2` String, `stage` String, `hints` String, `block_value` String, `remove_ad` String, `count` String, `action` String, `skin_id` String, `prv_skin_id` String, `date` String, `time_05s` String, `day` String, `text` String, `placement` String, `place` String, `networkName` String, `creativeId` String, `orientation` String, `mode` String, `time_1s` String, `line_item_1` String, `line_item_2` String, `type` String, `viewCount` String, `id` String, `reason` String, `connection` String, `screen` String, `app` String, `time_01s` String, `statusCode` String, `errorCode` String, `button` String, `eu` String, `issue` String, `per_user` String, `per_session` String, `interval_renamed` String, `ignore_in_a_row` String, `seq_num` String, `mopub_ilrd_adgroup_id` String, `mopub_ilrd_adgroup_name` String, `mopub_ilrd_adgroup_priority` String, `mopub_ilrd_adgroup_type` String, `mopub_ilrd_adunit_format` String, `mopub_ilrd_adunit_id` String, `mopub_ilrd_adunit_name` String, `mopub_ilrd_country` String, `mopub_ilrd_currency` String, `mopub_ilrd_id` String, `mopub_ilrd_network_name` String, `mopub_ilrd_network_placement_id` String, `mopub_ilrd_precision` String, `mopub_ilrd_publisher_revenue` String, `rewarded` String, `session` String, `applies` String, `lat` String, `personalized_ads` String, `consent_ads_state` String, `consent_easy_state` String, `option_state` String, `pages` String, `region_detection` String, `link` String, `banner_request` String, `banner_loaded` String, `banner_failed` String, `inter_request` String, `inter_loaded` String, `inter_failed` String, `rewarded_request` String, `rewarded_loaded` String, `rewarded_failed` String, `name` String, `level_progress` String, `state` String, `level_id` String, `from` String, `hints_unlock` String, `setting` String, `change_count` String, `tasks` String, `level_number` String, `content_version` String, `download_error` String, `middle` String, `value` String, `balance_version` String, `skip` String, `clickTrackingUrl` String, `test` String, `group` String, `fromScrn` String, `fromCtrl` String, `slideOn` String, `slideTo` String, `monthDif` String, `by` String, `dayDif` String, `month` String, `progress` String, `available` String, `userLvl` String, `cnt` String, `year` String, `game_mode` String, `main` String, `daily` String, `consent_ccpa_state` String, `created_at_micro` UInt64, `source_file_name_for_debug` String, `timezone` String, `impression_id` String, `waterfall` String) ENGINE = Distributed('bigbase12', 'lz', 'braingame_android_shard_bigbase12_ets_events', sipHash64(adid))
это не та таблица, нужен DDL от shard  braingame_android_shard_bigbase12_ets_events
источник

PL

Piotr Liakhavets in ClickHouse не тормозит
Denny Crane (I don't work at Yandex (never did))
это не та таблица, нужен DDL от shard  braingame_android_shard_bigbase12_ets_events
CREATE TABLE lz.braingame_android_shard_bigbase12_ets_events (`created_at_dt` Date, `file_dt` Date, `file_hr` UInt8, `created_at` UInt32, `installed_at` UInt32, `adid` FixedString(16), `idfa` FixedString(16), `idfv` FixedString(16), `gps_adid` FixedString(16), `city` String, `country_subdivision` String, `postal_code` String, `session_count` String, `install_finish_time` String, `install_begin_time` String, `app_id` String, `app_name` String, `app_version` String, `app_version_raw` String, `network_name` String, `campaign_name` String, `adgroup_name` String, `creative_name` String, `label` String, `is_organic` String, `gclid` String, `click_referer` String, `android_id` String, `referrer` String, `search_term` String, `country` String, `device_name` String, `device_type` String, `os_name` String, `api_level` String, `sdk_version` String, `os_version` String, `revenue_usd` String, `match_type` String, `activity_kind` String, `event_name` String, `tracker` String, `fb_campaign_group_name` String, `fb_campaign_name` String, `fb_adgroup_name` String, `tweet_id` String, `twitter_line_item_id` String, `connection_type` String, `isp` String, `conversion_duration` String, `last_time_spent` String, `time_spent` String, `lifetime_session_count` String, `source` String, `balance_ver` String, `game_count` String, `time_s` String, `score` String, `block_count` String, `lines` String, `squares` String, `best_score` String, `miss_block0` String, `miss_block1` String, `miss_block2` String, `stage` String, `hints` String, `block_value` String, `remove_ad` String, `count` String, `action` String, `skin_id` String, `prv_skin_id` String, `date` String, `time_05s` String, `day` String, `text` String, `placement` String, `place` String, `networkName` String, `creativeId` String, `orientation` String, `mode` String, `time_1s` String, `line_item_1` String, `line_item_2` String, `type` String, `viewCount` String, `id` String, `reason` String, `connection` String, `screen` String, `app` String, `time_01s` String, `statusCode` String, `errorCode` String, `button` String, `eu` String, `issue` String, `per_user` String, `per_session` String, `interval_renamed` String, `ignore_in_a_row` String, `seq_num` String, `mopub_ilrd_adgroup_id` String, `mopub_ilrd_adgroup_name` String, `mopub_ilrd_adgroup_priority` String, `mopub_ilrd_adgroup_type` String, `mopub_ilrd_adunit_format` String, `mopub_ilrd_adunit_id` String, `mopub_ilrd_adunit_name` String, `mopub_ilrd_country` String, `mopub_ilrd_currency` String, `mopub_ilrd_id` String, `mopub_ilrd_network_name` String, `mopub_ilrd_network_placement_id` String, `mopub_ilrd_precision` String, `mopub_ilrd_publisher_revenue` String, `rewarded` String, `session` String, `applies` String, `lat` String, `personalized_ads` String, `consent_ads_state` String, `consent_easy_state` String, `option_state` String, `pages` String, `region_detection` String, `link` String, `banner_request` String, `banner_loaded` String, `banner_failed` String, `inter_request` String, `inter_loaded` String, `inter_failed` String, `rewarded_request` String, `rewarded_loaded` String, `rewarded_failed` String, `idfa__hashed` UUID ALIAS CAST(coalesce(UUIDNumToString(MD5(hex(MD5(hex(MD5(nullif(UUIDNumToString(idfa), '00000000-0000-0000-0000-000000000000'))))))), '00000000-0000-0000-0000-000000000000'), 'UUID'), `idfv__hashed` UUID ALIAS CAST(coalesce(UUIDNumToString(MD5(hex(MD5(hex(MD5(nullif(UUIDNumToString(idfv), '00000000-0000-0000-0000-000000000000'))))))), '00000000-0000-0000-0000-000000000000'), 'UUID'), `gps_adid__hashed` UUID ALIAS CAST(coalesce(UUIDNumToString(MD5(hex(MD5(hex(MD5(nullif(UUIDNumToString(gps_adid), '00000000-0000-0000-0000-000000000000'))))))), '00000000-0000-0000-0000-000000000000'), 'UUID'), `android_id__hashed` UUID ALIAS CAST(coalesce(UUIDNumToString(MD5(hex(MD5(hex(MD5(nullif(lower(android_id), ''))))))), '00000000-0000-0000-0000-000000000000'), 'UUID'), `name` String, `level_progress` String, `state` String, `level_id` String, `from` String, `hints_unlock` String, `setting` String, `change_count` String,
источник

PL

Piotr Liakhavets in ClickHouse не тормозит
Denny Crane (I don't work at Yandex (never did))
это не та таблица, нужен DDL от shard  braingame_android_shard_bigbase12_ets_events
 `tasks` String, `level_number` String, `content_version` String, `download_error` String, `middle` String, `value` String, `balance_version` String, `skip` String, `clickTrackingUrl` String, `test` String, `group` String, `fromScrn` String, `fromCtrl` String, `slideOn` String, `slideTo` String, `monthDif` String, `by` String, `dayDif` String, `month` String, `progress` String, `available` String, `userLvl` String, `cnt` String, `year` String, `game_mode` String, `main` String, `daily` String, `consent_ccpa_state` String, `created_at_micro` UInt64, `source_file_name_for_debug` String, `timezone` String, `impression_id` String, `waterfall` String) ENGINE = MergeTree PARTITION BY (created_at_dt, file_dt) ORDER BY (event_name, adid, created_at) SETTINGS index_granularity = 8192
источник

U

Ugly in ClickHouse не тормозит
/report
источник

U

Ugly in ClickHouse не тормозит
!report
источник

U

Ugly in ClickHouse не тормозит
да как тут))
источник

U

Ugly in ClickHouse не тормозит
некогда нам, у нас тут насущные проблемы
источник

DC

Denny Crane (I don't... in ClickHouse не тормозит
Piotr Liakhavets
 `tasks` String, `level_number` String, `content_version` String, `download_error` String, `middle` String, `value` String, `balance_version` String, `skip` String, `clickTrackingUrl` String, `test` String, `group` String, `fromScrn` String, `fromCtrl` String, `slideOn` String, `slideTo` String, `monthDif` String, `by` String, `dayDif` String, `month` String, `progress` String, `available` String, `userLvl` String, `cnt` String, `year` String, `game_mode` String, `main` String, `daily` String, `consent_ccpa_state` String, `created_at_micro` UInt64, `source_file_name_for_debug` String, `timezone` String, `impression_id` String, `waterfall` String) ENGINE = MergeTree PARTITION BY (created_at_dt, file_dt) ORDER BY (event_name, adid, created_at) SETTINGS index_granularity = 8192
21060207-20200603

ааа я понял, (created_at_dt, file_dt), это два разных поля, это не легаси наименование парта

>Partition value mismatch between two parts with the same partition ID.
>Existing part: 21060207-20200603_66880_121440_2570, newly added part: 21060207-20200603_1350_1350_0.

т.е. это вообще не то что я подумал, ясно. Это просто авторинкремент в зукипере с номером блока внезапно стал 1350 хотя уже был 121440.
источник

ИИ

Илья Ионов in ClickHouse не тормозит
Всем привет! Ребят, внезапно перестались логироваться запросы в query_log. Никто не сталкивался случайно? Клик не ребутали, конфиги не переччитывали
источник

DC

Denny Crane (I don't... in ClickHouse не тормозит
Илья Ионов
Всем привет! Ребят, внезапно перестались логироваться запросы в query_log. Никто не сталкивался случайно? Клик не ребутали, конфиги не переччитывали
конфиг сам перечитывается, КХ следит за изменением файла
источник

ИИ

Илья Ионов in ClickHouse не тормозит
>не переччитывали
всмысле файлы не трогались
источник

DT

Dmitry Titov in ClickHouse не тормозит
Илья Ионов
Всем привет! Ребят, внезапно перестались логироваться запросы в query_log. Никто не сталкивался случайно? Клик не ребутали, конфиги не переччитывали
select * from system.settings  WHERE name='log_queries';
источник

ИИ

Илья Ионов in ClickHouse не тормозит
Я на это и смотрю) Там 0, changed тоже 0. Но в конфигах все ок и запросы до недавнего времени залетали
источник

DT

Dmitry Titov in ClickHouse не тормозит
ну, эта настройка может стоять и у профиля пользователя и у конкретного запроса.
источник