Skip to content

Instantly share code, notes, and snippets.

@marksteele

marksteele/gist:c2002b44b005ea1fda63

Created March 13, 2015 18:29
Show Gist options
  • Save marksteele/c2002b44b005ea1fda63 to your computer and use it in GitHub Desktop.
Save marksteele/c2002b44b005ea1fda63 to your computer and use it in GitHub Desktop.
Download ZIP
Another riak crash
Raw
gistfile1.txt
curl -v http://10.19.10.23:8098/buckets/lead_duplicate_email/keys?keys=true
* About to connect() to 10.19.10.23 port 8098 (#0)
* Trying 10.19.10.23... connected
> GET /buckets/lead_duplicate_email/keys?keys=true HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: 10.19.10.23:8098
> Accept: */*
>
2015-03-13 18:27:53.455 UTC [error] <0.29303.789> gen_fsm <0.29303.789> in state initialize terminated with reason: bad arithmetic expression in riak_core_coverage_plan:create_plan/5 line 59
2015-03-13 18:27:53.455 UTC [error] <0.29303.789> CRASH REPORT Process <0.29303.789> with 0 neighbours exited with reason: bad arithmetic expression in riak_core_coverage_plan:create_plan/5 line 59 in gen_fsm:terminate/7 line 611
2015-03-13 18:27:53.456 UTC [error] <0.476.0> Supervisor riak_kv_keys_fsm_sup had child undefined started with {riak_core_coverage_fsm,start_link,undefined} at <0.29303.789> exit with reason bad arithmetic expression in riak_core_coverage_plan:create_plan/5 line 59 in context child_terminated
@marksteele
Copy link
Author

On 1.4.12-1

@marksteele
Copy link
Author

2015-03-13 18:17:51 UTC =ERROR REPORT====
** State machine <0.24411.789> terminating
** Last event in was timeout
** When State == initialize
* Data == {state,undefined,riak_kv_keys_fsm,{state,{raw,21126305,<0.24372.789>}},<<"4">>,riak_kv,all,1,{riak_kv_listkeys_req_v4,<<"lead_duplicate_email">>,none},21126305,undefined,0,infinity,riak_kv_vnode_master,#Fun<riak_core_coverage_fsm.2.78317664>,#Fun<riak_core_coverage_fsm.4.78317664>}
* Reason for termination =
* {badarith,[{riak_core_coverage_plan,create_plan,5,[{file,"src/riak_core_coverage_plan.erl"},{line,59}]},{riak_core_coverage_fsm,initialize,2,[{file,"src/riak_core_coverage_fsm.erl"},{line,218}]},{gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,494}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2015-03-13 18:17:51 UTC =CRASH REPORT====
crasher:
initial call: riak_core_coverage_fsm:init/1
pid: <0.24411.789>
registered_name: []
exception exit: {{badarith,[{riak_core_coverage_plan,create_plan,5,[{file,"src/riak_core_coverage_plan.erl"},{line,59}]},{riak_core_coverage_fsm,initialize,2,[{file,"src/riak_core_coverage_fsm.erl"},{line,218}]},{gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,494}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]},[{gen_fsm,terminate,7,[{file,"gen_fsm.erl"},{line,611}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
ancestors: [riak_kv_keys_fsm_sup,riak_kv_sup,<0.413.0>]
messages: []
links: [<0.476.0>]
dictionary: [{'**DTRACESUPPORT',unsupported}]
trap_exit: false
status: running
heap_size: 987
stack_size: 24
reductions: 1026
neighbours:
2015-03-13 18:17:51 UTC =SUPERVISOR REPORT====
Supervisor: {local,riak_kv_keys_fsm_sup}
Context: child_terminated
Reason: {badarith,[{riak_core_coverage_plan,create_plan,5,[{file,"src/riak_core_coverage_plan.erl"},{line,59}]},{riak_core_coverage_fsm,initialize,2,[{file,"src/riak_core_coverage_fsm.erl"},{line,218}]},{gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,494}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
Offender: [{pid,<0.24411.789>},{name,undefined},{mfargs,{riak_core_coverage_fsm,start_link,undefined}},{restart_type,temporary},{shutdown,5000},{child_type,worker}]

2015-03-13 18:27:28 UTC =ERROR REPORT====
** State machine <0.29093.789> terminating
** Last event in was timeout
** When State == initialize
* Data == {state,undefined,riak_kv_keys_fsm,{state,{raw,69751234,<0.3573.789>}},<<"4">>,riak_kv,all,1,{riak_kv_listkeys_req_v4,<<"lead_duplicate_email">>,none},69751234,undefined,0,infinity,riak_kv_vnode_master,#Fun<riak_core_coverage_fsm.2.78317664>,#Fun<riak_core_coverage_fsm.4.78317664>}
* Reason for termination =
* {badarith,[{riak_core_coverage_plan,create_plan,5,[{file,"src/riak_core_coverage_plan.erl"},{line,59}]},{riak_core_coverage_fsm,initialize,2,[{file,"src/riak_core_coverage_fsm.erl"},{line,218}]},{gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,494}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2015-03-13 18:27:28 UTC =CRASH REPORT====
crasher:
initial call: riak_core_coverage_fsm:init/1
pid: <0.29093.789>
registered_name: []
exception exit: {{badarith,[{riak_core_coverage_plan,create_plan,5,[{file,"src/riak_core_coverage_plan.erl"},{line,59}]},{riak_core_coverage_fsm,initialize,2,[{file,"src/riak_core_coverage_fsm.erl"},{line,218}]},{gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,494}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]},[{gen_fsm,terminate,7,[{file,"gen_fsm.erl"},{line,611}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
ancestors: [riak_kv_keys_fsm_sup,riak_kv_sup,<0.413.0>]
messages: []
links: [<0.476.0>]
dictionary: [{'**DTRACESUPPORT',unsupported}]
trap_exit: false
status: running
heap_size: 987
stack_size: 24
reductions: 1026
neighbours:
2015-03-13 18:27:28 UTC =SUPERVISOR REPORT====
Supervisor: {local,riak_kv_keys_fsm_sup}
Context: child_terminated
Reason: {badarith,[{riak_core_coverage_plan,create_plan,5,[{file,"src/riak_core_coverage_plan.erl"},{line,59}]},{riak_core_coverage_fsm,initialize,2,[{file,"src/riak_core_coverage_fsm.erl"},{line,218}]},{gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,494}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
Offender: [{pid,<0.29093.789>},{name,undefined},{mfargs,{riak_core_coverage_fsm,start_link,undefined}},{restart_type,temporary},{shutdown,5000},{child_type,worker}]

2015-03-13 18:27:53 UTC =ERROR REPORT====
** State machine <0.29303.789> terminating
** Last event in was timeout
** When State == initialize
* Data == {state,undefined,riak_kv_keys_fsm,{state,{raw,130165387,<0.21559.789>}},<<"4">>,riak_kv,all,1,{riak_kv_listkeys_req_v4,<<"lead_duplicate_email">>,none},130165387,undefined,0,infinity,riak_kv_vnode_master,#Fun<riak_core_coverage_fsm.2.78317664>,#Fun<riak_core_coverage_fsm.4.78317664>}
* Reason for termination =
* {badarith,[{riak_core_coverage_plan,create_plan,5,[{file,"src/riak_core_coverage_plan.erl"},{line,59}]},{riak_core_coverage_fsm,initialize,2,[{file,"src/riak_core_coverage_fsm.erl"},{line,218}]},{gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,494}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2015-03-13 18:27:53 UTC =CRASH REPORT====
crasher:
initial call: riak_core_coverage_fsm:init/1
pid: <0.29303.789>
registered_name: []
exception exit: {{badarith,[{riak_core_coverage_plan,create_plan,5,[{file,"src/riak_core_coverage_plan.erl"},{line,59}]},{riak_core_coverage_fsm,initialize,2,[{file,"src/riak_core_coverage_fsm.erl"},{line,218}]},{gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,494}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]},[{gen_fsm,terminate,7,[{file,"gen_fsm.erl"},{line,611}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
ancestors: [riak_kv_keys_fsm_sup,riak_kv_sup,<0.413.0>]
messages: []
links: [<0.476.0>]
dictionary: [{'**DTRACESUPPORT',unsupported}]
trap_exit: false
status: running
heap_size: 987
stack_size: 24
reductions: 1026
neighbours:
2015-03-13 18:27:53 UTC =SUPERVISOR REPORT====
Supervisor: {local,riak_kv_keys_fsm_sup}
Context: child_terminated
Reason: {badarith,[{riak_core_coverage_plan,create_plan,5,[{file,"src/riak_core_coverage_plan.erl"},{line,59}]},{riak_core_coverage_fsm,initialize,2,[{file,"src/riak_core_coverage_fsm.erl"},{line,218}]},{gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,494}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
Offender: [{pid,<0.29303.789>},{name,undefined},{mfargs,{riak_core_coverage_fsm,start_link,undefined}},{restart_type,temporary},{shutdown,5000},{child_type,worker}]

@marksteele
Copy link
Author

riak-admin status

1-minute stats for 'riak@10.19.10.23'

riak_kv_stat_ts : 1426272162
vnode_gets : 0
vnode_gets_total : 14657
vnode_puts : 0
vnode_puts_total : 11574
vnode_index_refreshes : 0
vnode_index_refreshes_total : 0
vnode_index_reads : 0
vnode_index_reads_total : 0
vnode_index_writes : 0
vnode_index_writes_total : 0
vnode_index_writes_postings : 0
vnode_index_writes_postings_total : 0
vnode_index_deletes : 0
vnode_index_deletes_total : 0
vnode_index_deletes_postings : 0
vnode_index_deletes_postings_total : 0
node_gets : 1
node_gets_total : 12155
node_get_fsm_siblings_mean : 0
node_get_fsm_siblings_median : 0
node_get_fsm_siblings_95 : 0
node_get_fsm_siblings_99 : 0
node_get_fsm_siblings_100 : 0
node_get_fsm_objsize_mean : 0
node_get_fsm_objsize_median : 0
node_get_fsm_objsize_95 : 0
node_get_fsm_objsize_99 : 0
node_get_fsm_objsize_100 : 0
node_get_fsm_time_mean : 0
node_get_fsm_time_median : 0
node_get_fsm_time_95 : 0
node_get_fsm_time_99 : 0
node_get_fsm_time_100 : 0
node_puts : 0
node_puts_total : 5500
node_put_fsm_time_mean : 0
node_put_fsm_time_median : 0
node_put_fsm_time_95 : 0
node_put_fsm_time_99 : 0
node_put_fsm_time_100 : 0
read_repairs : 0
read_repairs_total : 0
coord_redirs_total : 6070
executing_mappers : 0
precommit_fail : 0
postcommit_fail : 0
index_fsm_create : 0
index_fsm_create_error : 0
index_fsm_active : 0
list_fsm_create : 0
list_fsm_create_error : 0
list_fsm_active : 0
pbc_active : 14
pbc_connects : 0
pbc_connects_total : 3031
late_put_fsm_coordinator_ack : 0
node_get_fsm_active : 0
node_get_fsm_active_60s : 0
node_get_fsm_in_rate : 0
node_get_fsm_out_rate : 0
node_get_fsm_rejected : 0
node_get_fsm_rejected_60s : 0
node_get_fsm_rejected_total : 0
node_put_fsm_active : 0
node_put_fsm_active_60s : 0
node_put_fsm_in_rate : 0
node_put_fsm_out_rate : 0
node_put_fsm_rejected : 0
node_put_fsm_rejected_60s : 0
node_put_fsm_rejected_total : 0
leveldb_read_block_error : undefined
riak_pipe_stat_ts : 1426272162
pipeline_active : 0
pipeline_create_count : 0
pipeline_create_one : 0
pipeline_create_error_count : 0
pipeline_create_error_one : 0
cpu_nprocs : 168
cpu_avg1 : 3
cpu_avg5 : 8
cpu_avg15 : 13
mem_total : 4147150848
mem_allocated : 795914240
disk : [{"/",41283904,3},
{"/dev",2020868,1},
{"/run",809992,1},
{"/run/lock",5120,0},
{"/run/shm",2024976,0}]
nodename : 'riak@10.19.10.23'
connected_nodes : ['riak@10.19.10.24','riak@10.19.10.25','riak@10.19.10.26',
'riak@10.19.10.27']
sys_driver_version : <<"2.0">>
sys_global_heaps_size : 0
sys_heap_type : private
sys_logical_processors : 2
sys_otp_release : <<"R15B01">>
sys_process_count : 1263
sys_smp_support : true
sys_system_version : <<"Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:2:2] [async-threads:64] [kernel-poll:true]">>
sys_system_architecture : <<"x86_64-unknown-linux-gnu">>
sys_threads_enabled : true
sys_thread_pool_size : 64
sys_wordsize : 8
ring_members : ['riak@10.19.10.23','riak@10.19.10.24','riak@10.19.10.25',
'riak@10.19.10.26','riak@10.19.10.27']
ring_num_partitions : 128
ring_ownership : <<"[{'riak@10.19.10.23',26},\n {'riak@10.19.10.24',26},\n {'riak@10.19.10.25',26},\n {'riak@10.19.10.26',25},\n {'riak@10.19.10.27',25}]">>
ring_creation_size : 128
storage_backend : riak_kv_multi_backend
erlydtl_version : <<"0.7.0">>
riak_control_version : <<"1.4.12-0-g964c5db">>
cluster_info_version : <<"1.2.4">>
riak_search_version : <<"1.4.12-0-g7fe0e00">>
merge_index_version : <<"1.3.2-0-gcb38ee7">>
riak_kv_version : <<"1.4.12-0-gc6bbd66">>
sidejob_version : <<"0.2.0">>
riak_api_version : <<"1.4.12-0-gd9e1cc8">>
riak_pipe_version : <<"1.4.12-0-g986a226">>
riak_core_version : <<"1.4.10">>
bitcask_version : <<"1.6.8-0-gea14cb0">>
basho_stats_version : <<"1.0.3">>
webmachine_version : <<"1.10.4-0-gfcff795">>
mochiweb_version : <<"1.5.1p6">>
inets_version : <<"5.9">>
erlang_js_version : <<"1.2.2">>
runtime_tools_version : <<"1.8.8">>
os_mon_version : <<"2.2.9">>
riak_sysmon_version : <<"1.1.3">>
ssl_version : <<"5.0.1">>
public_key_version : <<"0.15">>
crypto_version : <<"2.1">>
sasl_version : <<"2.2.1">>
lager_version : <<"2.0.1">>
goldrush_version : <<"0.1.5">>
compiler_version : <<"4.8.1">>
syntax_tools_version : <<"1.6.8">>
stdlib_version : <<"1.18.1">>
kernel_version : <<"2.15.1">>
memory_total : 142392648
memory_processes : 14053280
memory_processes_used : 14053256
memory_system : 128339368
memory_atom : 602729
memory_atom_used : 593574
memory_binary : 1222888
memory_code : 12982992
memory_ets : 2292000

@marksteele
Copy link
Author

root@kv01:~# curl -v http://10.19.10.23:8098/buckets/lead_duplicate_email/props

  • About to connect() to 10.19.10.23 port 8098 (#0)
  • Trying 10.19.10.23... connected

GET /buckets/lead_duplicate_email/props HTTP/1.1
User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
Host: 10.19.10.23:8098
Accept: /

< HTTP/1.1 200 OK
< Vary: Accept-Encoding
< Server: MochiWeb/1.1 WebMachine/1.10.0 (never breaks eye contact)
< Date: Fri, 13 Mar 2015 18:50:45 GMT
< Content-Type: application/json
< Content-Length: 485
<

  • Connection #0 to host 10.19.10.23 left intact
  • Closing connection #0
    {"props":{"allow_mult":true,"backend":"bitcask_mult_lead_duplicate_email","basic_quorum":false,"big_vclock":50,"chash_keyfun":{"mod":"riak_core_util","fun":"chash_std_keyfun"},"dw":"quorum","last_write_wins":false,"linkfun":{"mod":"riak_kv_wm_link_walker","fun":"mapreduce_linkfun"},"n_val":"4","name":"lead_duplicate_email","notfound_ok":true,"old_vclock":86400,"postcommit":[],"pr":0,"precommit":[],"pw":0,"r":"quorum","rw":"quorum","small_vclock":50,"w":"quorum","young_vclock":20}}

@marksteele
Copy link
Author

oot@kv01:~# riak attach
Remote Shell: Use "Ctrl-C a" to quit. q() or init:stop() will terminate the riak node.
Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:2:2] [async-threads:0] [kernel-poll:false]

Eshell V5.9.1 (abort with ^G)
(riak@10.19.10.23)1> redbug:start("riak_core_coverage_plan:create_plan").
ok
quitting: timeout
(riak@10.19.10.23)2>

@marksteele
Copy link
Author

(riak@10.19.10.23)2> redbug:start("riak_core_coverage_plan:create_plan").
ok

15:09:28 <<0.17818.790>> {riak_core_coverage_plan,create_plan,
[all,<<"4">>,1,129636575,riak_kv]}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment