Inputs page fails on RHEL7 and derivatives

[cheskyherskovic]

Hi when splunk server doesn't have outside connection the inputs page on this addon doesn't work i've tested it on

system info RedHat 7.9
attached logs for this issue
cisco_intersight_logs.txt

[jerewill-cisco]

Thanks @cheskyherskovic

I had one other person contact me about being unable to access the inputs page but they were not able to share much and I wasn't ever able to reproduce it. I will take a look at your logs and try to reproduce the issue myself once again. The inputs page is provided by the Splunk Add-on Builder. I can't think of anything that I'm doing that would cause this kind of problem, but if it's related to the server not having outside connectivity then that's a pretty key bit of data that I didn't have before. Thanks for the report.

[jerewill-cisco]

Hi @cheskyherskovic,

It's taken me a bit of time to properly test your idea and I don't think that outside connectivity is the issue. But I do have a new theory that I'd like for you to test?

Can you please try again and confirm that the Splunk user that you are using to configure the inputs for the Add-on has the admin_all_objects capability in one of their roles? I believe I may have been able to reproduce the issue by using a Splunk user that does not have this capability in their RBAC.

If this does turn out to be the root cause, I believe it is related to how the Add-On builder uses the secret storage. See here for more details...

https://dev.splunk.com/enterprise/docs/developapps/manageknowledge/secretstorage/secretstoragerbac

[cheskyherskovic]

Comfirmed user had admin_all_objects i will try to replicate it again with out internet and I'll let you know my results

[cheskyherskovic]

@jerewill-cisco I've successfully isolated the issue, and interestingly enough it's not the internet that's the issue the issue is centos 7 I've repeated this with 2 brand new installations on centos/RedHat 7 centos 8 it works fine it's problem initializing the modular inputs I've attached the splunkd.log
splunkd.log

[cheskyherskovic]

After digging a little more it seems that the issue is that the cryptography library is not compatible with CentOS/RHEL 7 since it requires glibc 2.18

[jerewill-cisco]

I had not tested any centos/rhel 7. That's certainly a different problem! Thanks for your help @cheskyherskovic. I'm going to be traveling for the next few weeks, but I'll try to get a centos7 dev box up and running in January.

[jerewill-cisco]

I was able to test today on CentOS 7 as well as Amazon Linux 1... both exhibit the same behavior. I think based on common distributions these should be the only two with glibc so old as to have this problem. Given the age of RHEL7 (and downstreams) at this point I think I'm okay with calling this a no-fix?

[cheskyherskovic]

@jerewill-cisco i agree with that. I'll be working to upgrade my box to RHEL 8 soon thanks for the work

[jerewill-cisco]

I really appreciate your help @cheskyherskovic. This same issue did occur with at least one other person already and it’s very nice to have the root cause even if it doesn’t get fixed. If you have further feedback, please feel free to share it via an issue or discussion.