Render view inside block with caching

My setup is a bit unconventional. I have a view displaying a block, relying on a contextual filter (let's call it product ID). I also have a custom block that renders this view programmatically because I need to include this block in multiple places on the page and I have some custom logic that pulls the actual product ID to call the view with. Basically:

$view = Views::getView('view_id'); 
$args = ['product_id' => $whatever_product];
return $view->buildRenderable('views_block_id', $args);

The process basically works but, as usual, I'll have problems when there are several such blocks on the page. Views only caches using the block id as a cache tag, so the first rendered view gets cached and displayed in all places. Naturally, switching off the cache would work:

return $view->buildRenderable('views_block_id', $args, FALSE);

but not exactly what I have in mind, I don't want to lose the benefits of caching.

My initial thought was quite simple, let's use custom cache tags in the view, thanks to views_custom_cache_tag. So I did, including the argument from the contextual filter:

views_block:view_id-views_block_id
custom:{{ arguments.product_id }}

But it still doesn't work.

Is there any other way I missed? I can't push new cache tags right before I try to render the view. The usual view hooks don't get called in this case (the second block already gets the cached variant, without even bothering to go near the hooks).