ManageIQ
Implement a near real-time refresh for ManageIQ that provides OpenStack #23589
Unanswered
lucky-sideburn
asked this question in
Q&A
Replies: 8 comments 27 replies
-
|
We are also facing a similar challenge with near real-time refresh, and I noticed that the current ManageIQ documentation does not specifically cover this use case. From earlier discussions with the MIQ team, my understanding is that if the Event Catcher is connected, enabled, and properly configured, any captured event should trigger a targeted refresh of the corresponding resource in the MIQ database. @ManageIQ Team — if possible, could you kindly list all the prerequisites and configuration steps required in MIQ to ensure immediate refresh for new resources? Many users seem to encounter similar issues, so it would be helpful to confirm whether this feature is working as designed and that our challenges are due to configuration mismatches — or if there are any known issues/bugs in this area. Having this documented in the official MIQ docs would be extremely valuable for the community. |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
Yeah, we can add something in the docs around refresh frequency. I'm wondering if maybe that belongs in the capabilities matrix? @agrare , what do you think? |
Beta Was this translation helpful? Give feedback.
-
|
Dear @Fryguy this can be very useful for us. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Oh how strange - I wrote up a whole response here a few days ago on my phone and it's not here 😭 . Seems I never hit send 🤦 |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Near real time is a function of what the provider itself supports. For example in VMware and Kubernetes they have an API where they stream changes to us, and those changes are very granular. There's are roughly 3 types of refreshes. The worst is just scheduled, which is we do a full refresh on a schedule. Next, a little better, if events are available we do event based refreshes, which could be full or targeted depending on the event content. The best is streaming is that API is available as described above. Right now open stack is the middle one (@agrare keep me honest here). You can see the refresh event handlers here. I'm not sure if openstack itself supports that kind of streaming API... If so, we can definitely move towards the streaming refresher which is the near real time one. If open stack does not have that kind of streaming API, that's probably what you'll have to implement first in upstream openstack. |
Beta Was this translation helpful? Give feedback.
4 replies
-
|
Thanks for the clear explanation, @Fryguy . For OpenStack, we are trying to perform targeted refreshes using the AMQP service. @agrare has helped me troubleshoot some issues earlier, though they weren’t fully resolved. So I request to list all the required configurations, verify they are correctly applied, and then perform testing. Once done, we plan to document the process so it will be useful for everyone. |
Beta Was this translation helpful? Give feedback.
-
|
Openstack definitely has targeted refresh I can confirm that. There is a separate endpoint that is required to be configured for events, currently supported values include: AMQP, Ceilometer, and STF. https://www.manageiq.org/docs/reference/latest/managing_providers/cloud_providers/openstack_providers.html#adding-openstack-providers covers the Events endpoint and how to configure ceilometer to store events. Based on the history of the project most of the documentation/terminology is closely tied to Red Hat OpenStack Platform but the intent was never to only work on RHOSP. This is the list of events that are handled https://github.com/ManageIQ/manageiq-content/tree/master/content/automate/ManageIQ/System/Event/EmsEvent/OpenStack.class It is entirely possible this list is out of date or different event names are raised for AMQP-based events and we can update for that accordingly. @lucky-sideburn / @dhamo-ror the first thing I would check is that you have the Event endpoint configured properly Next confirm that you have an EventCatcher worker running, you can go to Application Settings / Diagnostics / Workers Or on your worker appliances check Once we know that the worker is running we need to see if we're getting any events from the OpenStack system. You're going to want to enable debug logging and check the log output for the event catcher. Application Settings / Settings / Logging You should see something like: If you've gotten that far and still aren't seeing targeted refreshes going through likely the event types are missing in our content repo and we'd need to update them. Let me know if you run into any snags doing this and I'll help out |
Beta Was this translation helpful? Give feedback.
-
|
Hi @agrare, thank you for your inputs, I'll test the mentioned things and update you tomorrow! |
Beta Was this translation helpful? Give feedback.
-
|
Thank you @agrare |
Beta Was this translation helpful? Give feedback.
-
|
Hi All, I’ve been working on enabling OpenStack events in my ManageIQ production environment. I followed the documented steps to configure the event catcher, and I can now see that events are being queued and processed. Here are some sample logs from my environment: From what I can see: 1.) The event catcher (ManageIQ::Providers::Openstack::CloudManager::EventCatcher) is picking up the OpenStack events. 2.) Events are being queued (MiqQueue.put) and then dequeued for processing. 3.) EmsEventHelper#before_handle confirms that the event is being processed. So, it looks like the event pipeline is working in production. How can I enable targeted refresh for OpenStack in my production ManageIQ? I already have the OpenStack provider added and events seem to be coming in. Do I need to change any settings, or does targeted refresh work automatically once events are received? |
Beta Was this translation helpful? Give feedback.
10 replies
-
|
Oh sorry yes you're right |
Beta Was this translation helpful? Give feedback.
-
|
That is extremely helpful @dhamo-ror so resource_id being nil is why there are no targets parsed. I'll start looking at this now |
Beta Was this translation helpful? Give feedback.
-
|
I've opened ManageIQ/manageiq-providers-openstack#921 to fix the issue with the network event payload but I would like to test the other types. If possible could you get the payloads for floatingip, router, port, subnet, security_group, and security_group_rules? |
Beta Was this translation helpful? Give feedback.
-
|
Also I'm curious if the CloudManager also has the same issue so if you could create an instance and get the |
Beta Was this translation helpful? Give feedback.
-
|
@agrare, I’ll test it and share the update with you tomorrow and we can discuss it then |
Beta Was this translation helpful? Give feedback.
-
|
Hi @agrare as per the discussion above, we merged the PR and tested the results, here the results are, network_id key and values are added, but it isn't updated in the db entries. |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Thanks @SanthoshRam98 can you try running some of the commands from above? And can you post some logs from around the time that event came in specifically looking for queued ems refreshes |
Beta Was this translation helpful? Give feedback.
-
|
Hi @agrare, |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Awesome thank you @SanthoshRam98 so it looks like the updated event target parser is working, before |
Beta Was this translation helpful? Give feedback.
-
|
Yes @agrare its working now, i modify the file little bit and now it's fine, file: app/models/manageiq/providers/openstack/network_manager/event_target_parser.rb is this file is ok? And what we can do for the volumes? |
Beta Was this translation helpful? Give feedback.
7 replies
-
|
And what is the process of targeted refresh for volumes @agrare. |
Beta Was this translation helpful? Give feedback.
-
|
@SanthoshRam98 that looks very similar to what I put up last week ManageIQ/manageiq-providers-openstack#921 did that not work for you? It seems like the only functional difference is where I'm getting the resource_id here you're also checking for If that is something you see in your event payloads I can add that, but can you post the event payload that requires that change? Thanks! |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
but the volume events are not capturing @agrare, we have any modifications right here? the logs when i created volume in the openstack, |
Beta Was this translation helpful? Give feedback.
-
|
@SanthoshRam98 so far I've only seen examples of network event payloads. If you can get me a storage manager/volume event payload and also a compute event payload I can fix the other event target parsers. |
Beta Was this translation helpful? Give feedback.
-
|
Dear @agrare, I’m a colleague of @lucky-sideburn and I’d like to provide an update with some additional details from our side regarding our ManageIQ Radjabov-1 setup. We are currently running ManageIQ Radjabov-1, and OpenStack Epoxy, in a dedicated docker-compose environment used for testing and development. Environment DetailsOur setup includes the following core containers:
Provider ConfigurationBelow is our configuration for the OpenStack provider connection using RabbitMQ (AMQP) inside :workers:
:worker_base:
:event_catcher:
:defaults:
:poll: 0.5.seconds
:poll_method: normal
:ems_event_page_size: 200
:ems_event_thread_shutdown_timeout: 30.seconds
:memory_threshold: 3.gigabytes
:heartbeat_freq: 15.seconds
:event_catcher_openstack_service: "auto"
:event_catcher_openstack:
:count: 2
:poll: 0.3.seconds
:amqp_hostname: "dashboard.xaaslab.it"
:amqp_username: "manageiq_ro"
:amqp_password: "Password"
:amqp_port: 5672
:ems:
:ems_openstack:
:excon:
:omit_default_port: true
:read_timeout: 180
:ems_refresh:
:openstack:
:is_admin: true
:heat:
:is_global_admin: false
:inventory_object_refresh: true
:allow_targeted_refresh: true
:openstack_network:
:is_admin: true
:inventory_object_refresh: true
:allow_targeted_refresh: true
:cinder:
:is_admin: true
:inventory_object_refresh: true
:allow_targeted_refresh: trueAfter configuring everything, I added the OpenStack provider in ManageIQ, validated both the provider and AMQP connection, and confirmed they were working correctly. Then I created a Cloud Tenant (with tenant mapping enabled) — the corresponding OpenStack Project was successfully created as expected. EventCatcher Issue and TroubleshootingThe EventCatcher worker doesn’t seem to function properly. We confirmed that the AMQP connection and event publishing from OpenStack are working correctly by successfully testing with a standalone Python consumer, ruling out any configuration issues on the OpenStack side. In the RabbitMQ logs, I can see it connects periodically and immediately disconnects, without transferring any data: Furthermore, an analysis with tcpdump confirmed that traffic packets are only exchanged during the initial MIQ connection, validation, and provider addition. After that, we see no subsequent traffic, which strongly supports the theory that the EventCatcher worker is failing and closing the connection immediately. From the ManageIQ logs ( The error suggests the EventCatcher starts but fails because the The provider is correctly configured and visible from the Rails console: ManageIQ::Providers::Openstack::CloudManager.all
=> [#<ManageIQ::Providers::Openstack::CloudManager id: 12, name: "devstack", tenant_mapping_enabled: true, enabled: true>]However, the EventCatcher worker is not found when checking the workers table, confirming the immediate termination: MiqWorker.where(:type => "ManageIQ::Providers::Openstack::CloudManager::EventCatcher").pluck(:id, :status, :pid, :queue_name)
=> []It looks like the worker cannot correctly fetch the OpenStack provider ID during startup. We can also provide screenshots of the Web GUI showing the data input for adding the provider and the provider status page, if that helps in the diagnosis. Could you please confirm whether this might be caused by a misconfiguration, or if there’s a known issue with the provider ID lookup in Radjabov-1, especially in a containerized/docker-compose environment? Thanks again for your help and time, |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
Just to clarify, you're using the monolithic docker version? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @Fryguy, Thanks for the quick response and for asking for clarification! Yes, to confirm: we are using the monolithic Docker version ( However, we have also performed tests with the Appliance version, and the result is identical: the EventCatcher worker attempts to establish the AMQP connection but is terminated immediately. This strongly suggests the issue is not related to the containerization environment but likely to an anomaly in the worker's initialization logic within Radjabov-1 when connecting to our OpenStack. Technical Issue and Strategic GoalsThe core issue remains the immediate failure of the EventCatcher worker during startup. This termination is confirmed by:
Our Objective and Key QuestionsOur goal is to implement near real-time synchronization (Targeted Refresh) from OpenStack to ManageIQ, as frequent full provider refreshes are not sustainable in production. We aim to use the native EventCatcher capability. Provider Event Bus → EventCatcher Worker → MiqQueue → EventHandler → Automate We kindly ask for your insight on the following points, starting with basic functionality:
AMQP Configuration SummaryFor your reference, here is the relevant configuration from :workers:
:worker_base:
:event_catcher:
# ... (Defaults omitted for brevity)
:event_catcher_openstack_service: "auto"
:event_catcher_openstack:
:count: 2
:poll: 0.3.seconds
:amqp_hostname: "dashboard.xaaslab.it"
:amqp_username: "manageiq_ro"
:amqp_password: "Password"
:amqp_port: 5672
:ems_refresh:
:openstack:
:is_admin: true
:inventory_object_refresh: true
:allow_targeted_refresh: true
# ... (Other services omitted)Thank you again for your time. Your guidance in diagnosing this core startup issue is highly valued. — Nick |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Dear ManageIQ Team,
I am working for an important customer in the telecom sector. We are currently using ManageIQ (Petrosian version, planning to upgrade to Radjabov) and OpenStack (Epoxy version).
Could you please advise on the best way to implement a near real-time refresh? We would like to minimize the delay for new resources created in OpenStack.
To clarify, at the moment we do not plan to have too many providers. We will have two OpenStack instances managed by ManageIQ, each with approximately 100 VMs (ranging from 1 to 10 VMs per instance).
Thank you in advance for your guidance.
Best Regards,
Eugenio M.
Beta Was this translation helpful? Give feedback.
All reactions